Some notes on applying computational divided differencing in optimization

We consider the problem of accurate computation of the finite difference f(x+ s)− f(x) when ‖s‖ is very small. Direct evaluation of this difference in floating point arithmetic succumbs to cancellation error and yields 0 when s is sufficiently small. Nonetheless, accurate computation of this finite difference is required by many optimization algorithms for a “sufficient decrease” test. Reps and Rall proposed a programmatic transformation called “computational divided differencing” reminiscent of automatic differentiation to compute these differences with high accuracy. The running time to compute the difference is a small constant multiple of the running time to compute f . Unlike automatic differentiation, however, the technique is not fully general because of a difficulty with branching code (i.e., ‘if’ statements). We make several remarks about the application of computational divided differencing to optimization. One point is that the technique can be used effectively as a stagnation test. 1 Finite differences Many nonlinear optimization routines require a sufficient decrease test on an iterate, which involves computation of a finite difference. For example the Armijo (also called “backtrack”) line-search requires testing an inequality of the form f(x+ αp)− f(x) ≤ σα∇f(x)p, which is called a “sufficient decrease” condition. The trust region method involves evaluating a ratio of the form f(x+ s)− f(x) m(x+ s)−m(x) in which the numerator is a finite difference of the objective while the denominator is a finite difference of a quadratic model. See Nocedal and Wright [2] for more information about these the Armijo line-search and trust region method. ∗Department of Combinatorics and Optimization, University of Waterloo, 200 University Ave. W., Waterloo, Ontario, Canada, N2L 3G1. Supported in part by an NSERC Discovery grant and a grant from the U.S. Air Force Office of Scientific Research.